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Abstract: This paper reveals a new evaluation model, which enables educational program and project 
managers to evaluate their programs with a simple and easy to understand approach. The “index of success 
model” is comprised of five parameters that enable to focus on and evaluate both the implementation and results 
of an educational program. The integration of these parameters forms a complete and comprehensive picture, 
which in turn provides the most information on the success or failure of the program. The parameters are: (1) the 
relevance index, referring to an evaluation of predefined objectives; (2) the efficiency index, referring to the fiscal 
efficiency of the program; (3) the effectiveness index evaluating the attainment of the program’s objectives; (4) 
the impact index gauging the effect on the consumers, for whom the program is undertaken; and (5) the final 
parameter is the program’s sustainability index. Since Israel’s education strategy currently leans towards 
site-based monitoring and evaluation, such a model can potentially be of great help to school management and 
staff. 
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1. Introduction 

How many times, after having completed a program or project, have you asked whether or not it was fruitful? 
You may feel content, and even be commended by a superior, but is this enough? Or, do we need clear indicators 
by which to judge whether or not our objectives have been met? 

The suggested model’s strength resides in its simplicity, as it allows any project manager — even one who is 
not a professional assessor — to construct specific measures of success. Applying this model requires a clear 
definition of objectives and targets, as well as a coherent decision-making process, and these vital topics will be 
examined as well. 

This work is divided into two main sections. The first section takes a look at evaluation procedures generally 
and the evaluation of programs and projects specifically, which includes a review of relevant core terms such as 
vision, policy, objectives, targets, programs and project. The second part of the paper outlines the “index of 
success model”, and includes operative explanations for implementing the model, as well as its pedagogical 
significance. 

2. Literature review 

2.1 What is evaluation? 

Traditionally, evaluation within the education system is hierarchical — the “expert” superior evaluates his/her 
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subordinate. Generally, a teacher evaluates a student, a principal evaluates a teacher, and an inspector evaluates a 
principal. Since the evaluation is determined according to the superior’s preference, and for the greater part 
focuses on the subordinate’s achievements and results, it can be regarded as arbitrary. Recently an awareness has 
grown for the need to systematize the field of evaluation through recognizing how, when, and what to evaluate, 
and for what purpose. 

It is common to find the words “measurement” and “evaluation” used in tandem within the education system. 
This points to a preconception that measurement is needed for a proper evaluation, and raises the question: What 
is measurement and what, in the context of the education system, needs to be measured? In order to answer this 
question, it will first be taken a look at what evaluation is. A number of researchers have uniformly concluded that 
evaluation is an act that provides a relevant opinion (Nevo, 2001; Freidman, 2006; Rossi, et al., 1999). 

As a statement of opinion, an evaluation can be either subjective or objective. Anyone looking out a window 
on a rainy day, and observing rays of sun forming a rainbow in the distance, will likely state that “there is a 
rainbow in the sky”. Likewise, when people say “car”, they all refer to a transportation vehicle. These are 
instances of an objective evaluation. Evaluating a work of art in a museum, on the other hand, may result in as 
many differing opinions and even there are critics, in what may be termed subjective evaluation. 

It is commonly assumed that objective evaluation is founded on systematic measurement of one procedure or 
another, hence the joining of the terms “measurement and evaluation” in education. The rationale for objectifying 
such an evaluation is the ability, through measurement, of comparing it to predefined standards. These standards 
are a yardstick by which to judge the data obtained in the evaluation. In comparison, subjective evaluation relies 
on intuition and feeling. 

It is the opinion of this author that any evaluation, even a scientific one accepted in such fields as the social 
and physical sciences, contains within it both subjective and objective elements. Let us take a study as an example 
that examines the mathematical achievements of fourth graders. The results of 1,500 fourth graders, selected 
according to sample study guidelines, are collected. This describes quantitative (vs. qualitative) research. Typical 
of such studies, this one reports findings such as the percentage of students achieving high scores, low scores and 
so on. Since the report is a result of statistical calculations, it is considered to be objective. 

The research paper, however, normally includes a discussion that attempts to interpret the results. This 
interpretation constitutes a subjective evaluation, as it is possible for two researchers to provide two different 
analyses for the same results. That is, when attempting to explain why the achievements of one group of fourth 
graders are different from another’s, the interpretation will be subjective. One researcher may claim that 
achievements of students living in a rural region are higher than those of students living in urban regions, because 
calmer surroundings contribute to the students’ grades vis-a-vis their wellbeing. Another may claim that rural 
results are higher because parents are more likely to take a greater role in their children’s education. 

The research findings are the same (objective) and the interpretations different (subjective), indicating the 
presence of both in a quantitative study. The fact is truer in a qualitative study. Nevertheless, there is a prevalent 
agreement that the process is an analytical one. 

In order to better elucidate the forthcoming model, this work will adopt Weis’s (1988, p. 4) definition of 
evaluation: the systematic examination of the implementation or results of a program or project, including a 
comparison with hidden and evident standards, for the purpose of improving or deciding upon said program or 
project. This definition is helpful since it allows to treat evaluation as multidirectional and multidimensional, not 
simply as an assessment of achievement and results. Weis’s (1988, p. 4) definition allows to evaluate the entire 
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educational spectrum, including objectives, planning, process, results and the environment and culture in which 
these programs take shape. The prevalent view in today’s educational system is that each and every dimension of 
life in an educational organization is worthy of being evaluated. Unlike in the 1960s, when students alone were 
evaluated, now the entire staff, educational programs and organizational features are similarly subject to 
monitoring (Nevo, 2001). 

The main emphasis of evaluation is a systematic analysis with three main objectives (Nevo, 2001; Freidman, 
2006): 

(1) To provide information on the degree to which the program or action conforms to expectations or 
requirements. If during the evaluation it is found that procedures are not followed as prescribed, alternatives 
should be offered; 

(2) To facilitate the detection of program flaws and the creation of corresponding solutions; 

(3) To examine the results of procedures and actions in order to decide on future action — continuing, 
terminating or altering the program. 

As described above, evaluation is a process undertaken during or post project, defined by the terms coined by 
Scriven (1967) as formative evaluation (during) and summative evaluation (post). The former begins with the 
project, lasts for its duration, and is meant to offer progressive improvement. The latter is performed at the 
conclusion of a program and measures its success or failure based on data accumulated during its implementation. 

Evaluation serves as a genuine tool for the betterment of school organizations, the promotion of best 
practices and the reliable attaining of objectives. These demand systematic and proven procedures, the use of 
research-based and needs-based evaluation tools and, more importantly, an integration of an appropriate outlook 
in the surveying and surveyed organizations. 

Evaluation professionals rightly claim that a systematic survey of an educational program can and should be 
carried out by qualified persons who are familiar with social science research methods (Rossi, et al., 1999). 
Recommendation 1.10 of the Dovrat Report (presented by the national taskforce for the promotion of education in 
Israel, 2005) reads: 

Responsible administration on the part of schools and kindergartens demands conformance to defined standards, and 
periodic measurement of achievement of the school and of the student. Measurement and evaluation must be performed 
professionally and sensitively, while incorporating internal and external evaluation tools. The purpose of evaluation is the 
continued improvement of educational and scholastic achievement in its myriad forms, without compromising the 
organization’s autonomy. 

External evaluation can be delegated to professional evaluators. However, when it comes to the educational 
organization’s internal examination, simpler tools that can be handled by non-professionals are called for. Thus, 
this model allows for measuring success by means of the internal capacity of the school. 

Two additional concepts related to evaluation procedures are feedback and monitoring. Feedback is a 
reaction to an act or its consequence and the actual reporting and supply of information collected and or at work in 
the evaluation process. It can be either a stage in the evaluation or a description of the entire process. Feedback 
can be communicated verbally or in writing, and in both cases serves as a summary of findings from a process. It 
is a mistake to call the completion of questionnaires after a certain activity “feedback”, since the forms are a part 
of the evaluation process itself. It would be more correct to state that the questionnaires will allow the examiner to 
give feedback on this or that activity. The feedback will consist of the results processed from the filled-out forms, 
which are handed over to the examiner or another consumer. In the same fashion, the act of collecting information 
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(e.g., form filling) cannot be considered feedback. However, when we summon a subordinate and provide him/her 
with an opinion of his/her performance, his/her role within the organization is provided with feedback. It follows 
that after feedback is provided, decisions are made. After decisions are made, monitoring processes must be 
established in order to guarantee that they are being properly carried out. 

Prior to describing the model, it is necessary to clarify the relationships between vision, policy, objectives, 
targets, programs and projects. 

2.2 The relationships between some factors in the model 

2.2.1 From vision to program 

The relationship between vision, policy, goals and objectives is hierarchical and, as one moves downwards, 
the terms become increasingly practical and applicable. The apex of the scale is occupied by vision, followed by 
policy, goals and ending with objectives. The hierarchical significance is also quantitative. That is, each policy can 
have a few goals, which could be translated into several objectives. After objectives are delineated, a program that 
will include several projects can be drawn, as illustrated in Figure 1 . 




Figure 1 From vision to objectives and developing a work plan 



2.2.2 Vision and policy 

The difference between vision and policy is not self-evident. Vision is the anchor and central idea around 
which an organization functions (Levi, 2000). It is the organization’s pole-star, or a dream that will never be 
completely realized, but is definitely aimed towards realization. 

Organizational vision should be expressed in one or a few simple sentences that are easy to remember: Take 
Walt Disney’s vision as an example: “the happiest place on earth”; or, similarly, Nokia’s motto of “connecting 
people”; vision is a dream to be realized. 

Does vision always accompany policy? Must there be a defined vision in order to establish policy? Although 
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the answer is not clear-cut, most of the theoreticians concur that there must indeed exist such a vision. Academics 
have tried to define “policy” for years, and some have shown a preference for using the term “mission” instead 
(Levi, 2000). Most dictionaries choose to define policy as the way of managing certain affairs. Although such a 
definition is simple and useful, other more involved alternatives have been suggested. Taylor (1997) raised several 
of them: 

(1) Policy is a weave of decisions and action. It is an authoritative allocation of resources; 

(2) Policy is an explicit or implicit description of a purposeful action taken to solve identifiable problems. As such, 
policy is a position adopted with respect to a particular conflict or specific goal; 

(3) Policy is composed of two elements: one is political decisions of which values resources will be spent on 
according to the organization's goals; the other is the measure that will be taken towards the fulfillment of these 
decisions. 

More recent definitions have been suggested, such as Freidman’s (2006) defining policy as fundamental 
guidelines that determine goals and objectives. It is a basic point of view of a particular sphere, adopted for the 
sake of solving problems and improving processes. 

It is possible to summarize the mentioned definitions and say that policy is general guidelines for certain 
activities that, when performed, fulfill the policy. The policy defines the core program of an organization, or its 
central operative sphere. It is common for organizations to associate vision with policy. Those for whom it is 
separate, tend to express vision in a short sentence and extrapolate policy in detail. As an example, we can look at 
the following educational vision: “To nurture a mature, independent and creative thinker capable of facing the 
challenges of the 21st century”. From this brief statement, it can be extrapolated the following policy: “An 
educational system that recognizes children’s varied developmental capabilities, based on area of study and age, 
and leads to a readiness for successfully exiting the system”. In order for the vision-based policy to be fulfilled, an 
organization must set goals and objectives for itself. 

2.3 Goals 

Goals are the substance of what a person or an organization desires to achieve. They describe the measures to 
be adopted in order to fulfill a given policy or similarly, how the policy is put into practice. Goals relate to the 
aspirations, purpose and vision. There is no unique form in which to present them, but unanimous opinion holds 
that they should be framed in positive terms. A goal should make clear what the desired outcome is, and it is thus 
the opinion of this author that it should always start with a “To ...” that indicates a particular action. “To teach 
through varied means” or “To identify in 2008 the specific needs of second graders” are instances of goals. 

The advantage of framing goals in this manner is that the substantive reason for taking the action is implicit 
in the statement. Framing a goal should answer the question “What for?” or “For whom?”. The answer is “in order 
to” or “for the purpose of’, and therefore the desired framing must describe the reason for the action, which 
demands the use of “to” followed by a verb. 

2.4 Objectives 

It is easy to confuse goals and objectives, since they both aim at achieving a desired outcome. Objectives are 
the battle plan, the stepping-stones on the path towards the achievement of the goal. They are derived from goals, 
and usually each goal includes several objectives. A most convenient approach is the SMART model developed 
and popularized by Stephen Covey (1990) as a tool to help people set and reach their goals. SMART stands for 
specific, measurable, achievable, realistic and time -bound. 

(1) Specific: Specific means that the objective is concrete, detailed, focused and well defined. Specific means 
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that it is results- and action-orientated. An objective must be straight forwards and emphasize action and the 
required outcome. They need to be straightforward, well defined and to communicate what you would like to see 
happen. One should avoid setting unclear or vague objectives, and therefore should be as precise as possible. If 
the goal was “to do well in my exams”, one of the specific objectives should be “to increase study time to 30 
hours per week and attend all scheduled revision sessions”. 

(2) Measurable: If the objective is measurable, it means that the measurement source is identified and we are 
able to track the actions as we progress towards the objective. Measurement is the standard used for comparison. 
It has to be clear how one will recognize when he/she has achieved his/her objective. Using numbers, dates and 
times is one way to represent clear objectives. If the goal was “to get physically fit”, making it measurable will be 
“to swim 20 lengths, twice a week this month”. 

(3) Achievable: Objectives need to be achievable, if the objective is too far in the future, it could be difficult 
to keep motivated and to strive to attain it. Objectives, unlike aspirations and visions, need to be achievable to 
keep one being motivated. Setting unfeasible objectives will only end in disappointment. Most objectives are 
achievable but, may require a change of priorities to make them happen. Objectives have to be challenging, but 
realistic. 

(4) Realistic: Objectives that are achievable may not be realistic. However, realistic does not mean easy. 
Realistic means having the resources to get it done. The achievement of an objective requires resources, such as, 
skills, money, equipment, etc. 

(5) Time -bound: Time -bound means setting a deadline for the achievement of the objective. Deadlines need 
to be both achievable and realistic. If time will not be set, motivation and urgency required to execute the tasks, 
would be reduced. 

2.5 Program 

After defining the function of goals and objectives, one could construct an educational program that will 
enable us to reach them. A program is an organized and detailed description of the actions needed in order to 
achieve a set of goals and objectives, in the pursuit of a given policy (Freidman, 2006). A program contains four 
main components: (1) goals, which the program seeks to accomplish; (2) means, the tools and resources available 
for implementing the program; (3) actions need to be taken 1 ; and (4) expected outcome after the program has been 
carried out. These are important to note, since the evaluation process examines each of them. 

2.6 Project 

A project is a complex and singular task, with defined beginning and end dates, and is constructed of a 
number of interdependent actions to be completed within a set timeframe and with given resources. Every project 
has defined goals, objectives and desired outcomes. A familiar or repetitive procedure cannot be characterized as a 
project (Hobbs, 2002; Lewis, 1995). Any task or action in a project is called a component. 

It is possible to demonstrate the hierarchical nature of the terms above through the case of “developing young 
leadership” in schools. The vision wished to realize is “nurturing a creative, independent thinker who is capable of 
facing the challenges of the 21st century”. Figure 2 illustrates the hierarchical structure of policy, programs and 
projects that follows the statement of vision. 



1 This is known as a “task list” or “to do list” — A list of steps to be taken in project management. This can be though of as an 
inventory list that substitutes for memory. Retrieved from http://en.wikipedia.Org/wiki/Task_list#Task_list. 
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Figure 2 From policy to projects 



3. The index of success model 

The “index of success model” (henceforth, IOS model) is comprised of five parameters that allow us to focus 
on and evaluate both the implementation and the outcomes of a program. The integration of these parameters 
forms a complete and comprehensive picture, and provides the most information on the success or failure of any 
program. The parameters (or indices) of success gauge the program’s goals, objectives and efficiency, impact on 
the consumers for whom it is undertaken and, finally, its sustainability. 

Before describing the components of the model, it is first necessary to define two additional terms: efficiency 
and effectiveness. It is all too easy to confuse the two terms or even equate them. Efficiency is the relationship 
between effort and results. That is, efficiency describes the process of a given action. In evaluating a program in 
general, or an educational program in particular, it is common to estimate efficiency in terms of cost (monetary 
and resources). If a given process produces the desired results, the lower its cost, the more efficient it is 
considered to be. Effectiveness refers to the results of a program. If the desired results have been achieved, the 
process is considered to be effective. It is possible to imagine a process which is effective yet inefficient. Consider 
the following case: You are standing on one side of main street holding a five -pound parcel, and would like to 
move the package to the other side of the street. If you chose to use a forklift capable of moving five tons, and the 
package is moved from one side of the street to the other, you are bound to be effective, since the desired effect 
will surely be accomplished. But the process will also be highly inefficient since its cost will be high and 
disproportionate to desired outcome. It would be much simpler to use a zebra crossing and walk to the other side. 
Again, efficiency refers to the process while effectiveness refers to the result. 

Now that it have made clear all the necessary concepts the model uses, it is possible to describe the IOS 
model and its five indices, as described in Figure 3. 
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(after the 
project 
completion) 



Figure 3 The IOS model and its five indices 



( 1 ) The relevance index 

It examines whether goals that have been set before the start of a program, have been met or not. 
Acknowledging the achievement of goals takes place during the final stage of a program or project and is part of 
the program or project summary. Therefore, it should be considered as part of a summative evaluation. The 
question for this parameter to be answered is: Have we achieved our goals? If we have, then this index should 
register as full, and the organization can continue to act upon the same goals. Two factors may stand in the way of 
achieving this success: 

First, the goals are no longer relevant to the organization, hence the name of this index. The Jewish National 
Fund (JNF) can serve as an example. The organization’s original goal was to purchase land for the Jewish 
population in Israel. When there was no more land left to buy, the goal became obsolete, and JNF members had to 
reinvent their purpose. They came up with new goals of establishing industry for preparing, developing and 
preserving land. Two additional goals were reforestation and creating a water infrastructure. If the members of 
JNF had not managed to rethink their direction, the organization would have ceased to exist and had not lasted to 
its current age of 105. The goals that were set were not relevant any more, and therefore where revised. If goals 
are not relevant anymore, it is concerned as a legitimate procedure to redefine them or revise them, but it is crucial 
first to understand why the irrelevancy has occurred. 

Second, an ambiguous definition of the goal may lead to confusion within the organization and eventually to a 
program’s failure. Success depends on a clear, structured definition of goals. 

(2) The efficiency index 

It gauges to what degree a program or project has stayed within its projected budget. In case a program has 
veered beyond the budget, it is considered inefficient, and vice versa. This raises the question that, since efficiency 
refers to a process, why do these index measure monetary figures? Why can not budget conformance stand alone 
as a measurable objective? The efficiency index refers to budgetary concerns, since these explain a process which 
demands particular attention. In commercial organizations, which measure success by looking at costs and income, 
this index is of prime importance. An organization is successful when income exceeds costs. These days, with 
more and more schools becoming self-managed, they can be viewed as commercial enterprises with 
self-contained financial systems. This index should be considered as part of both formative and summative 
evaluations. Throughout the program, project managers should check the costs and expenses (formative 
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evaluation). If they find that they have acceded the budget, the program plan should be reconsidered in order to 
stay within the budget limits. As the program (or project) comes to an end, evaluation of the costs will be 
re-examined (summative evaluation). 

(3) The effectiveness index 

It measures to what degree objectives have been met. It is written in Figure 3 that this parameter measures if 
predefined inputs have been achieved. One should ask why inputs should be measured, because of effectiveness 
implies results or outputs? The answer is simple. Objectives are always established before a program. Therefore, 
they should be considered as inputs to the planning process. The program (or project) is actually based upon them. 
Thus the designed program serves as predetermined inputs. 

Managers should in fact investigate whether their programs were properly designed according to the set 
objectives. Reaching these objectives will indicate that the program is appropriate, and that so are the preset inputs. 
Baking a cake is a good example. We use a mixer to stir numerous ingredients. If the cake is a good one, it is 
obviously effective (i.e., successful), and it is also likely that all of the inputs (cake ingredients) were correct. If 
the cake is inedible, each of the inputs-ingredients must be examined to decide what to change to prevent a second 
failure. A program’s success implies that the inputs and processes used were all correct. If the managers do not 
reach their objectives, the program and its inputs must be reexamined. 

This index raises the question of whether or not all of the objectives have been met. If they have, then the 
managers can assume their program was a good one, and they can retain the same methods for the future. In case 
one or more objectives have not been attained, the managers must reexamine each objective’s program, and 
observe where they went wrong. How the program must change for the objectives to be met and become evident? 
Failing to reach certain objectives might mean that they are unrealistic or unattainable. The managers are prone to 
developing lofty expectations. It is therefore critical to discuss the viability of their objectives prior to setting them. 
Naturally, the managers must also refrain from safely setting objectives too low. 

This is where formative evaluation gains its importance. If the managers discover at certain points along the 
way that targets are unlikely to be met, they must promptly change their working program or define more realistic 
targets. Monitoring targets throughout a program allows them to make necessary changes. Nevertheless, they will 
ask themselves, as they summarize the results (summative evaluation), have they accomplished their objectives. 

(4) The impact index 

It provides a picture of how the program influences the consumer for whom it is implemented. It requires two 
actions: 

First, assessing the consumers’ expectation. Prior to initiating a program, the managers must ask the clients 
what they expect to achieve. By doing so, they also share their anticipations in what can be termed as an 
“expectation matching” stage. At this point, the managers can clarify to the program’s consumers which parts of 
the program will meet which goals and objectives, and which expectations will likely not be met. Matching 
expectations in this manner serves as a kind of contract between the clients and the evaluators and prevents future 
disappointments. After noting the consumers’ desires, the managers must return to them during the program 
(formative evaluation) and when it ends (summative evaluation). If the managers find that, while the program is in 
motion the clients expectations are not being met, they can add missing components to the program. The 
summative expectation evaluation should be performed with the clients, allowing them to explain if and why they 
failed to meet certain preset goals. 
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Second, assessing the consumers’ satisfaction. A formative evaluation of the clients’ satisfaction with the 
program’s progress should be made at least twice, as well as a summative one at the end. If the consumers are not 
happy with the managers’ progress, this measure will enable them to take remedial steps. If the summative 
evaluation confirms that the clients are fully satisfied then the managers can claim success with respect to this 
index, and vice versa. 

(5) The sustainability index 

This parameter does not always need to be figured into the IOS model. Since a project is a singular event, for 
instance, sustainability loses its meaning. On the other hand, if the managers intend to implement a program 
repeatedly, this index is of vital importance. Sustainability is gauged only during a summative evaluation. If the 
managers find, after the program is concluded, that the managers would like reuse or continue it, this index 
registers as successful. 

3.1 Measuring success in educational projects or programs 

During the planning stage of an educational projects or programs, one must determine what percentage of 
success will be assigned to each index. It is possible to arbitrarily assign each index 20%, as five of the total 100%, 
or any combination in between. There should be a good reason for assigning one index a greater percentage than 
another. If the managers decided that customers’ satisfaction is of prime importance in a certain project, they can 
assign the impact index 30% (i.e., 15% for matching expectations and the rest for client satisfaction). Certain 
indices may be absent from some programs. If, for instance, teachers wish to gauge their success after teaching a 
class, two parameters can be left out: efficiency, since a singular class does not normally entail special costs; and 
sustainability, since the lesson is over. Only the relevance, effectiveness and impact indices will be used in this 
case. The teachers can decide whether to assign each parameter an equal or different percentage. 

In order to observe how the model can be put to use, let us examine the following example of a new reading 
program, which is implemented for second grade students in a public school. They must first note the policy, then 
derive appropriate goals and objectives, and finally draw up a program to reach each of the latter. 

(1) School policy: The school acknowledges children’s varying developmental capacities, based on 
differences of age and area of study, and aims to prepare students for a successful exit from the educational 
system. 

(2) Goals that can be derived from policy: (a) to identify, within the 2008 academic year, the unique needs of 
children in the second grade; (b) to identify, for the same group and in the same period, difficulties in reading and 
writing; and (c) to raise within the 2008 academic year second graders’ reading and writing ability. 

Let us chose the third goal, for demonstrative purposes, and set objectives. It should be noted that teachers 
were to draw a real program, objectives must be set for each of the goals. 

(3) Objectives: (a) 90% of second graders will be able to read by the end of the 2008 academic year; (b) a 
reading and writing examination will be created and given twice, once in April and once in June, during the 2008 
academic year; and (c) 85% of the first and second grades’ teaching staff will undergo workshops of no less than 
28 cumulative hours. 

Let us now chose the first objective and construct a working program around it. As before, it should be noted 
that a real program will require action to be taken in response to each separate objective. Table 1 shows a possible 
program to be implemented during the first three months of school. 



10 




Measuring success: Evaluating educational programs 



Table 1 Working program 



September 2008 


October 2008 


November 2008 


Diagnosing all the students’ reading 
ability 


A test writing sample to gauge ability 


Adapting the personalized program 
based on results 


Creating personalized study programs 


A meeting with a well-known 
children’s author 


Exploring “writing a diary” 


A teacher-parent meeting to encourage 
reading habits 


Building a class library 


Children and parents read together 



Now that teachers have a program in place, they can measure its success according to the IOS model’s 
indices. The success of each index is gauged through a questioning technique. For simplicity, let us assign each 
parameter 20%. 

(1) The relevance index 

This index is examined at the end of the program or at the end of the 2008 academic year. The question is: 
Flave we raised second graders’ reading and writing abilities? If the answer is positive then it can be registered 
success for this index. If the students’ abilities have not been improved then the teachers have failed in this 
respect. 

(2) The efficiency index 

Let us assume, for the present purpose only, that the school has engaged the services of an external 
assessment organization to measure students’ reading ability. The external organization is also to process the 
numerical results of the tests. The school’s budget for this process is $3,000. In addition, the principal has received 
the local municipality’s authorization to employ a teacher’s assistant, at an annual cost of $2,000. The sum budget 
for the new program stands at $5,000. During the academic year, teachers will examine whether they are operating 
within the budget, or whether additional costs have been incurred in the form of an additional assistant, more tests, 
etc. If the budget is maintained, and no further changes are required, this index is registered as successful. If an 
additional assistant added $2,000 to the budget then this parameter is a failure. 

(3) The effectiveness index 

The teachers will ask three questions, one for each of their objectives. 

Ql: Can 90% of second grade students read by the end of the 2008 academic year? If the answer is yes, then 
this parameter registers as a success. If only 50% can read, then it has failed in this respect. This requires that the 
teachers evaluate reading ability throughout the year and make the necessary adjustments to their program. 

Q2: Has a reading and writing test been created, and was it administered once in April and once in June 
during the 2008 academic year? If so, then teachers are successful in this respect, and if not then the index 
registers as a failure. 

Q3: Have 85% of the teaching staff participated in 28 hours supplemental workshops? If so then it can 
register success, and vice versa. 

Achieving only some of the objectives will register as a partial success of this index. Two out of three, for 
instance, will denote a 67% success. 

(4) The impact index 

It demands the teachers to ask several questions. 

Ql: Have we matched our expectations with the consumers (in the case, the parents and students)? 

Q2: Have we continued to gauge the clients’ expectations throughout the program and at its end? 

Q3: Did we measure our customers’ satisfaction? 
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Q4: Are the customers content? 

Affirmative responses to these queries will indicate a success on this index, whereas negative answers will 
point to failure. 

(5) The sustainability index 

Will the program for improving students’ reading ability be extended into the 2009 academic year? If 
teachers decide that it will, this index will register as a success; while deciding to drop, the program will indicate 
failure. 

Now that the evaluation queries have been outlined, it is possible to summarize the progress in Table 2. 



Table 2 Summarize the evaluation process using the IOS model 



Index 


Remarks 


Percent of success 
(%) 


Relevance 


All goals were accomplished: We were able to raise the reading level of second 
graders. 


20 


Efficiency 


We did acceded budget limitations. We had to hire an extra assistant. 


0 


Effectiveness 


We achieved all of our objectives. 


20 


Impact 


(a) We coordinated acceptations but we did not check at the end of the program if 
we stood up to these acceptations. 


5 


(b) We examined the level of satisfaction during ad at the end of the program. All 
clients were satisfied. 


10 


Sustainability 


The program extended into the 2009 academic year. 


20 


Sum 




75 



It can therefore reach a sound estimate that the success in the second graders’ reading program has been 75%. 
All that remains are to decide what must be changed in order for the program to reach 100% the next time the 
teachers implement it. 

Table 3 offers a presentation of each stage in the model, enabling the teachers to closely follow the program’s 
progression. Teachers can thus manage the program’s information as it accumulates and changes. Naturally, the 
program’s coordinator can decide to enlarge or downsize parts of this chart, based his/her assessment of their 
importance. 



Table 3 Goals, objectives, work program and IOS 





Goals 




Objectives 


Preparation 
of work 
plan 


IOS 










Relevance 


Efficiency 


Effectiveness 


Impact 


Sustainability 


1 






















1 


















2 


















3 


















4 














2 






















1 


















2 


















3 















There is one final issue to clarify, and that is decision-making. Is it appropriate to make individual or group 
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decision in such a model? Let us first take a look at several examples of individual and collective 
decision-making. 

3.2 Decision-making 

A simplistic way of describing decision-making is choosing among alternatives. There are two reference 
groups of decision-making: personal and collective. Although the same considerations apply to both, personal 
decisions are made by an individual while collective ones are made by a group of people (Tartar & Hoy, 1998). 
This does not represent a dichotomy within an organization. Rather, the manager can decide which decisions are 
to be made collectively and which by him/her alone, as detailed in Figure 4. 



The manager 


The manager 


The manager 


The manager 


The manager 


The manager 


The manager 


decides by 


decides by 


arises a topic 


arises a topic 


arises a topic 


arises a topic 


holds a group 


himself and 


himself and 


and asks each 


for a staff 


for a staff 


for a staff 


discussion. 


does not 


informs the 


staff member 


discussion 


discussion, 


discussion. 


and the staff 


inform the 


staff 


for his or her 


and then 


asks them to 


The staff 


as one agrees 


staff 


members. 


opinion 


decides by 


recommend a 


decides 


upon the 


members. 




discussion 
and then 
decides by 
himself. 


himself. 


decision. Then 
he adopts (or 
not) the 

decision. 


together but 
the manager 
has the right 
to veto. 


decision 
either by vote 
and/or 
consensus. 



1 2 3 4 5 6 7 



*- 



Personal decision Group decision 

Figure 4 Line of the continuum of processes of decision making 

3.2.1 Personal decision-making 

In this category, decisions are made by individuals. There are three main approaches to this form of decision: 
(1) The rational approach demands a careful examination of the relevant elements and the creation of a variety of 
solutions from which the decision is drawn (Koopman & Pool, 1990); (2) The intuitive approach relies solely on 
inner feelings for guidance through the decision making process (Aldag & Stearns, 1991; Wagner & Sternberg, 
1990); and (3) The accumulated experience approach is based on links, experience, intuition and imagination 
(Klein & Weick, 2001). 

3.2.2 Collective decision-making 

This form of decision-making relies on feedback from a number of people. Modern managerial attitudes 
embrace this approach, as it is seen to be a cornerstone of organizational function (Collins, 1997). There are four 
basic approaches to group decision-making: (1) the rational approach is similar to the one described for personal 
decisions; (2) in the political approach, decision-makers are guided by their needs and feelings (Maoz, 1990); (3) 
an additional approach emphasizes the organization’s directives and procedures (Hiller & Self, 2004); and (4) 
finally, the “trashcan approach” mixes together unrelated problems and solutions into a “bin”, as these arise 
(Cohen, et al., 1972). 

The decision-making process is vital to the IOS model, and asks to pay special attention to the choice 
between personal or group decisions. It should be pointed out that the group administering the program or project, 
will choose to use the decision-making model that enables it to make better decisions. 

The relationship between the index of success, the evaluation process and the appropriate decision making 
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process is summarized in Table 4. The right column notes the index of success. The middle column notes what 
kind of evaluation, summative or formative, is taking place, and the left column notes the desired decision making 
format, personal or group. 



Table 4 Decision making during the evaluation process 



The Index 


Evaluation process 


Type 


Decision-making 


Relevance 


Summative evaluation: 
One can evaluate if the 
goals had been met, only 
at the end of the process 


Group decision 


In order to decide whether the goal has been met, it is 
advised to enable the relevant group to collaborate. 


Efficiency 


Formative and 

summative evaluation 


Personal decision 


There are only two options: either you stay within 
budget limitation or you don’t. Therefore, the project 
manager can overlook the budget limits by himself, and 
there is no reason for a conference. 


Effectiveness 


Formative and 

summative evaluation 


Personal decision 


The objectives are clear and quantified so that there are 
only two options: either you reach your objectives or 
you don’t. Therefore, the project manager can overlook 
the objectives by himself, and there is no reason for a 
conference. 


Impact 


Formative and 

summative evaluation 


Group decision 


In order to evaluate if acceptations have been met and if 
the clients are satisfied, there is a need for the input of 
the clients. 


Sustainability 


Summative evaluation 


Group decision 


It is advised that this kind of decision will be reached by 
the forum that is connected with the program. 



4. The pedagogical implications of the model 

The IOS model allows anyone managing a program or project in general, or an educational one in particular, 
to implement evaluation, feedback and control procedures. Most of the indices used by the model do not require 
them to seek professional assistance. Defining goals and objectives and creating a detailed working plan, should 
be a routine part of a program manager’s duties. Like any other skills which demand exercise, so does this 
managerial technique. 

Out of the five indices described, it is recommended to consult a professional evaluator for the parameter 
gauging the impact of the program on the customers. This index demands writing up a satisfaction questionnaire, 
which is a task best left for a professional evaluator. At the same time, it is also possible to adapt preexisting 
questionnaires to new needs. It may turn out that when the first time an educator runs this model, it will seem 
arduous. If, for instance, a program has two central goals, each of them must produce three objectives, and each of 
these must originate a detailed work plan, all of which merge into the overall program. This is quite a bit of work, 
but good preparation will be served to increase the chances of the project’s success. Implementing the IOS model 
provides with an accurate evaluation of a program or project, and enables to securely ascertain whether it is a 
success or a failure. 

In today’s educational environment, which strives to keep monitoring processes internal, such a model can 
facilitate the workflow of school principals and staff. 
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